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The  RDECOM-TARDEC  Visual  Perception  Eaboratory  (VPE)  was  designed  to  evaluate 
the  detectability  of  camouflaged  vehicles  in  both  the  visual  and  infrared  spectrum,  assess 
computer  simulated  renderings  of  camouflaged  vehicles  and  evaluate  electro  optical 
cameras  and  systems  for  homeland  defense  applications. 

Modern  image  fusion  technologies  will  allow  the  US  Military  to  achieve  the  surveillance, 
security  and  detection  applications  needed  in  21st  century  warfare.  When  combining 
information  extracted  from  multiple  sources,  the  fused  result  will  provide  more  details, 
resulting  in  a  superior  battlefield  advantage. 

Some  of  the  potential  applications  of  image  fusion  and  3D  displays  for  homeland  defense 
are  listed  below.  The  VPE  is  actively  working  with  various  Homeland  Security  liaisons  to 
locate  programs  to  develop  and  apply  these  technologies. 

•  Concealed  weapon  and  mine  detection  using  sensor  fusion  and  edge  enhancement 

along  with  passive  infrared  and/or  millimeter-wave  multi-band  imagery. 

•  Close-proximity  wrap-around  fusion  vision  system  for  airplane  passenger  cabins 

terrorist  identification. 

•  Surveillance  vehicle  with  wrap  around  enhanced  vision  and  concealed  weapon 

detection  for  peace  keeping  operations. 

•  Aerial  unmanned  robotic  surveillance. 
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•  Millimeter-wave  (mm-wave)  using  passive  sensors  to  detect  hidden  objects  -  no 

irradiation  of  subject 

•  Integration  of  3-Dimensioanl  (3D)  visual  images  and  mm-wave  to  increase  visibility 

and  clarity  of  the  image 

•  3D  display  technology  could  be  deployed  for  use  at  existing  airport  search  stations, 

or  imbedded  in  walls;  the  technology  unobtrusively  scan  passengers  and  other 
airport  personnel 
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Fig.  1:  Visible  Image,  mm-wave  image,  and  fused  image  of  concealed  weapon  with  Fuzzy  Logic  (Top  Row: 
left  to  right  respectively);  Infrared,  visible,  and  fused  image  of  a  pedestrian  in  the  road  (Bottom  row:  left  to 

Right  respectively) 

The  same  algorithms  that  are  used  for  fusion  of  multiband  imagery  for  mine  detection  can 
also  be  used  for  the  purpose  of  concealed  weapon  detection.  Fig.  1  illustrates  how  fuzzy 
based  image  fusion  can  be  used  to  segment  areas  of  an  image  that  may  show  a  concealed 
weapon.  The  VPL  has  been  working  with  Wayne  State  Univ.  and  Ethereal  Technology  in 
Ann  Arbor  to  apply  auto  stereoscopic  technology  and  image  fusion  to  the  problems  of 
detecting  people  carrying  weapons,  also  with  3D  baggage  inspection  at  airports,  and 
crowd  surveillance  at  large  public  facilities.  The  auto  stereoscopic  display  as  it  exists  in 
the  VPL  is  shown  below  in  Fig.  2.  The  unique  and  important  features  of  this  display  is 


that  it  is  not  dependent  on  users  wearing  goggles  or  glasses  and  is  a  high  resolution 
display.  A  3D  image  is  formed  by  the  brain  fusing  a  left  and  right  image  as  it  does  from 
the  left  and  right  eye.  Using  image  fusion  in  concert  with  3D  displays  has  the  potential  to 
greatly  improve  the  detection  of  concealed  objects. 


3-Dimensional  Display  and  Stereoscopic  Imagery: 


With  the  current  elevated  concern  over  security,  VPL  engineers  have  been  investigating 
using  3D  displays  to  increase  the  through-put  scanning  efficiency  and  accuracy  of 
detection  of  concealed  explosive  devices  in  packages.  Fig.  2  demonstrates  the  prototype 
3D  display  that  can  project  3D  images  from  a  stereoscopic  pair  of  images.  A  pair  of  X- 
ray  images,  such  as  the  one  of  a  package  shown  below,  that  are  obtained  from  a 
conventional  X-ray  scanner  such  as  is  used  at  the  TACOM  receiving  dock,  can  be 
presented  in  3D  to  observers  without  the  use  of  special  viewing  goggles. 


It  has  been  estimated  that  failures  in  advanced  man-machine  systems  can  be  attributed  to 
“human  error”,  and  increasingly  to  loss  of  operator  situational  awareness  (SA),  and  may 
be  as  high  as  75%.  Three  dimensional  displays  will  increase  SA  by  allowing  us  to  see 


information  collected  by  sensors  in  a  natural  environment  as  opposed  to  planar  images. 
There  are  a  number  of  mirror  systems  which  form  a  real  three  dimensional  image  in 
space  of  an  object  in  a  position  which  is  well  separated  from  the  object.  One  of  the  major 
parts  of  the  stereoscopic  display  used  in  the  VPL  is  the  spherical  concave  mirror.  A 
spherical  mirror  can  magnify  the  size  of  the  object  being  displayed  as  shown  in  Fig.  2. 

There  is  a  need  for  technology  to  scan  packages  and  use  this  3D  system  to  reduce  illegal 
transport  of  weapons  and  potential  threats  on  U.S.  soil.  This  display  can  be  applied  to 
crowd  surveillance  and  screening  at  public  access  points.  In  these  times  of  increased 
concerns  about  terrorists  and  passengers  carrying  concealed  weapons  or  other  harmful 
items,  sensor  fusion  and  3D  displays  could  be  of  benefit  in  alerting  guards  to  potential 
dangerous  passengers.  Combining  sensor  fusion  with  a  3D  display  could  also  improve 
the  recognition  rate  of  guards  using  cameras  that  scan  crowds  for  people  that  are  listed  in 
a  known  terrorist  database.  We  believe  that  implementing  volumetric  visible  and  IR 
images  will  improve  the  recognition  rate,  because  it  will  provide  a  more  detailed  3D 
image  or  thermal  properties  versus  only  a  2D  visible  image  of  possible  terrorization. 


Panoramic  Image  Fusion  (PIF)  System: 


Fig.  3:  Concept  of panoramic  image  fusion  for  ground  vehicles;  Ford  Lincoln  Navigator  with  PIF  system 

(left  to  right  respectively) 

The  VPL  team  and  Ford  Laboratories  are  testing  and  beginning  the  technology  insertion 
of  a  vision  system  called  Panoramic  Image  Fusion  (PIF).  Applications  of  this  system 
range  from  peace-keeping  operations  in  Iraq,  homeland  defense,  surveillance  at  boarders, 
early  warning  passive  detection  of  threats,  as  well  as  medical  telepresence.  The  vision 
system  is  mounted  on  to  the  front  and  rear  of  the  vehicle  and  provides  day  and  night 
panoramic  imaging  by  fusing  the  visible  and  infrared  video  from  multiple  cameras  in 
real-time.  The  system  uses  commercial  off-the-shelf  (COTS)  video  cameras  and  imaging 
boards.  Combine  imaging  sensor  fusion  capability  with  the  ability  to  look  anywhere 
around  the  vehicle  and  you  have  what  the  Army  calls  situational  awareness.  Why  is  more 
than  one  type  of  sensor  required?  Because  infrared  cameras  give  increased  night  vision 
capability  and  visible  sensors  are  of  little  to  no  use  at  night  or  in  areas  with  poor  lighting. 
Even  in  the  well-lit  areas,  night  vision  technology  provides  a  benefit  to  safety  by  making 
it  easier  to  detect  camouflaged  insurgents.  In  daylight,  panoramic  image  fusion  can  be 
used  to  detect  an  individual  in  a  crowd  who  has  recently  fired  their  weapon,  due  to  the 
heat  generated  by  the  weapon  as  seen  through  clothes  by  the  imaging  sensors.  Fig.  1 


depicts  a  faux  scenario  of  a  heated  weapon  underlying  an  article  of  clothing  of  a  person 
surrounded  by  a  crowd  of  civilians. 

The  benefit  of  panoramic  image  fusion  to  the  crew  of  an  armored  vehicle  is  that  it 
provides  situational  awareness  prior  to  egressing  from  the  vehicle.  A  soldier  or  officer  in 
a  closed  vehicle  can  use  this  technology  to  pan  and  zoom  around  the  vicinity  of  the 
vehicle  and  therefore  have  knowledge  of  possible  locations  of  potential  combatants.  The 
major  benefit  is  that  the  scene  can  be  interpreted  much  more  quickly  and  accurately, 
thereby  increasing  the  survivability  of  the  soldier.  An  Example  of  the  image  fusion 
process  for  driving  safety  is  shown  in  Fig.  1,  where  a  pair  of  infrared  and  visible  road 
crossing  scenes  that  were  fused  using  the  method  of  Fuzzy  Fogic. 

VPL  Status: 

In  FY  04  the  VPF’s  research  aimed  at  homeland  defense  and  the  Global  War  On 
Terrorism  (GWOT)  was  aired  on  a  local  television  channel,  hosted  by  Ms.  Marcella  Fee. 
The  segment  discussed  the  panoramic  image  fusion  system  being  developed  for  tactical 
vehicles  as  well  as  a  novel  3D  imaging  technology  that  could  be  used  to  scan  crowds  at 
entrances  to  public  structures.  The  VPF  was  also  highlighted  on  the  History  Channel’s 
recent  series  on  Stealth  on  Fand  series.  The  use  of  the  lab  for  camouflage  assessment  was 
described  and  CAPT  Hetzel  showed  how  a  soldier  takes  a  camouflaged  vehicle 
perception  test  in  the  VPF. 

As  mentioned  in  the  key  research  areas  above,  the  VPF  has  developed  a  panoramic  image 
fusion  system  for  increasing  the  situational  awareness  of  soldiers  in  Iraq,  National 


Guardsmen  and  patrol  officers  at  home.  This  technology  has  been  shown  to  many  PM’s 
(Stryker,  LAV,  Tactical  Vehicle,  HET)  and  all  who  have  seen  it  want  it  on  their  vehicles 
for  in-theatre  testing.  The  only  problem  at  this  time  is  funding  to  supply  the  fleet  of 
vehicles  that  various  customers  are  asking  for. 

Maintaining  situational  awareness,  dynamic  surveillance,  and  target  development  is 
important  for  the  soldier.  This  task  will  be  accomplished  through  image  sensor 
acquisition,  data  fusion  and  3D  visualization.  The  implementation  of  the  integrated 
image  system  will  support  real-time  SA  for  Homeland  Defense  by  providing  images  of 
vehicles  with  IR  signatures. 
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